The invention relates to a method for controlling the position of an object to be displayed in virtual reality on a display device of a graphics unit by way of a human-machine interface, with one or more stationary electronic cameras recording one or more unique marking symbols located on a side of the object facing the camera(s), and the images being evaluated with a computer in accordance with image-processing methods to yield the spatial position of the object, which is transmitted to the graphics unit, the associated object being correspondingly displayed, in the predetermined position of the object, as virtual reality on the display device.
The invention further relates to an input device for executing the method.
The human manipulation of the spatial position of objects is significant for numerous technical fields. It is fairly difficult, however, for a person to simultaneously control such an object, e.g., an excavator, in several degrees of freedom. Nevertheless, a great need exists for this option, especially in computer graphics. Here, the user wishes to view simulated, i.e., virtual, objects in selected views on the monitor, and select or alter the desired view as simply as possible.
In the most extreme case, this means that three translational and three rotary degrees of freedom must be controlled simultaneously. To facilitate the mastering of this task, an individual requires a user-friendly human-machine interface. The interface should resolve complex behavior patterns of the person into control signals for the computer. In principle, any system that can detect the position and orientation of the human hand in and around all three spatial directions through measurements is suited for this task.
In this connection, an input device of a human-machine interface is known as xe2x80x9cSpace Mousexe2x80x9d (registered trademark). This input device is used to control three rotational and three translational degrees of freedom for 3-D computer graphics; the device has a cover, which is flexibly mounted to a base plate and can be pushed and rotated by hand in/around a desired spatial direction. The deflections are measured, and can be interpreted as movement commands. This device is already in widespread use in the sense of a speed control in computer graphics.
This input device, however, only permits deflections of a few millimeters or degrees. It is therefore suited as a control device for translational and rotational speeds, but not for positions and angles. Its use is limited to a small working area, and the user must familiarize himself with the device and practice using it.
Exoskeletons and hand controllers have been developed for controlling positions and orientations. Exoskeletons and hand controllers are extremely complex mechanical structures, and are therefore not considered for applications in which the movements of the operator""s hand are limited by mechanical parts.
The so-called Data Glove (or Power Glove) is an input device used in applications in the area of cyberspace and virtual reality. It comprises a glove that is provided with sensors and is connected via cables to a computer. The computer is programmed to react to finger and wrist movements. Thus, in a graphic computer simulation, virtual objects can be recorded and manipulated by a virtual hand. Gross hand movements (arm movements) are not detected, however. The Data Glove is thus not suitable for solving the problem.
Another known device is the so-called joystick, which is primarily used in games and simulations. It is a small, easy-to-handle input device comprising a movable, manually-operated control lever. The joystick serves in controlling objects in different directions. Usually, two degrees of freedom can be controlled, similarly to the computer mouse connected to more modern computers. Due to special designs, rotating the handle permits the control of a further degree of freedom. Generally, the available degrees of freedom are insufficient. Even in the applications in which they suffice, however, joystick operation stipulates a familiarization phase, similarly to the xe2x80x9cSpace Mousexe2x80x9d (registered trademark).
Tracking systems (by the Polhemus company) are also known; in these systems, a transmitter creates a magnetic field, and the receiver derives the transmitter""s position and orientation from the magnetic field. While electromagnetic trackers ascertain the position and the orientation, and would therefore be suitable for taking measurements, the receiver, which the operator holds in his hand, must be connected to the electronic evaluation unit by way of an awkward cable or a radio device. Furthermore, iron parts skew the measurement result. Finally, these systems are very expensive.
In an interactive method known from WO 98/13746, a person carrying a sword is a virtual object, which is displayed in a graphics unit and battles a likewise virtual opponent. In this case, characteristic parts of the person""s body (elbows, shoulders, knees, hips) are marked, as is the sword; these markings serve in establishing the location of corresponding points in the creation of the virtual image.
In this known method, the human-machine interface is formed by a plurality of stationary, electronic cameras, which record numerous unique marking symbols located on the side of the objectxe2x80x94namely, the person with the swordxe2x80x94facing the cameras. The images are then evaluated with a computer according to image-processing methods, the result being the spatial position of the person and sword, which is transmitted to the graphics unit. There, the associated person and sword are correspondingly displayed as virtual reality on the display device of the unit.
The person is thus not an operator holding a sword as an input medium. This setup is therefore not a human-machine interface that is actively operated by an operator""s hand. Instead, the sword-bearing person observed by the camera is virtually simulated in his entirety on the display device of the graphics unit, because the device known from WO 98/13746 is intended to lead a match with a likewise virtual opponent in a computer-controlled game or computer-controlled exercise.
EP 0 813 040 A2 discloses a system for virtual modeling of a physical system with movable and immovable parts. Here, at least two video cameras are used to detect identification symbols of objects. These detected identification symbols are associated with objects displayed on a monitor, so a modeled virtual reality of the physical system appears on the monitor. No input of position information with the aid of an input medium guided by an operator""s hand is provided in this system, however.
WO 98/00774 discloses a special embodiment of a scanner platform that is equipped with an optical detector, e.g., a video camera, and is intended to recognize identifications on physical objects, such as products, that are manually guided, in a first region, across a glass plate serving as a spatial reference. The video camera, which is disposed in a second region, namely beneath the glass plate acting as a spatial reference element, tracks the optical identifications of the objects guided across the glass plate, detects their signals and inputs them as input signals into a computer for evaluation and later display on a monitor. The object is only tracked in three degrees of freedom, namely relative to the position in the plane of the reference element and the orientation around the vertical.
It is the object of the invention to disclose a method and an input device for controlling the position of an object that is graphically displayed in virtual reality, with which an observation of markings of the virtually-displayed object by electronic cameras allows the position of the virtually-displayed object to be manually controlled by the operator without limitations of his hand movements, that is, neither by mechanical parts nor cables, permitting the most intuitive possible use.
According to the invention, which relates to a method of the type mentioned at the outset, this object is accomplished in that the spatial position of the virtually-displayed object is controlled by means of an input medium of the human-machine interface, which can be actively displaced and rotated by the operator""s hand, and is formed by a material model object of the object to be displayed, which is easy to handle and is provided with the marking symbols.
The method of the invention is distinguished by the fact that any operator can successfully select and alter the desired view immediately, that is, without any practice. The direct spatial relationship between the material model object forming the input medium and the simulated computer-graphics model allows the operator to readily associate the correct use of the system, and operate it without previous experience.
The mobility of the hand of the input-medium operator is not impeded by mechanical components or electrical lines. The model object provided with the marking symbols is completely passive, and requires neither power nor built-in parts.
In an advantageous modification of the method according to the invention, the marking symbols provided on the model object also contain an identification of the respective object to be displayed as virtual reality, and are likewise recorded with the camera(s) and evaluated according to data-processing methods, so model objects provided with different marking symbols can be distinguished and their individual positions can be indicated, and the result of this evaluation is transmitted to a graphics unit (graphics workstation, PC graphics), with the object that has been recognized by way of its identification and thereby associated being displayed as virtual reality on the display device in the position of the respective model object, as manually predetermined by the operator.
An input device for executing the method of the invention is distinguished by the fact that the marking symbols are located on the underside of the model object, which faces the camera(s).
Flat marking symbols are advantageously provided on the underside of the model object. The flat marking symbols applied to the underside of the model objects can be letters, numbers and/or abstract symbols, and can appear in color or be reflective so they can be better recognized and distinguished.
The symbols related to the identification of the object can also appear, for example, in the form of a barcode. The necessary marking symbols are simple to produce and apply, and are not apparent to the user, because they are located on the underside of the model object.
Marking symbols in different forms can be applied to the underside of a model, so different objects of virtual reality that are distinguished by the application of different identification symbols can be activated with a single model object.
In an advantageous input device according to the invention, the camera or cameras is or are located beneath an optically transparent plate, above which the operator holds the model object with its underside in the position facing the camera(s). The plate preferably comprises glass or Plexiglas (registered trademark) having one or more of the following properties: transparency or translucence, clearness or matte finish, coloring or absence of color, coating or no coating.
In an input device according to the invention, an electronic camera (e.g., video camera, CCD camera) having an interface that is suitable for electronic image transfer to a digital computer can be provided as a camera.
A personal computer can be used for image evaluation related to the position of the marking symbols, and thus of the model object, and the identification symbols of the model object. It is therefore possible, at low cost, to embody an input device for executing the method of the invention using components available on the mass market.
Particularly in the field of advertising, the method and the input device of the invention can advantageously be implemented to display advertised products in desired positions. An illustrative example cited here is the concept of a xe2x80x9cvirtual auto dealership.xe2x80x9d All of the vehicle types offered by a dealer are stored in different embodiments as computer graphics, and can be displayed on a monitor or a projection screen.
The method of the invention advantageously presents anyone, namely salespeople, management or customers, with the opportunity to select and vary the desired view of a selected vehicle type, without necessitating any experience.
In an embodiment of a xe2x80x9cvirtual auto dealershipxe2x80x9d possessing the features of the invention, one or more electronic cameras is or are disposed beneath an optically transparent plate. In the use of only one camera, the optical axis is preferably oriented perpendicular to the plate. In the use of at least two cameras, their optical axes can likewise be oriented perpendicular. (This is the classic stereo arrangement with parallel axes.) It is advantageous, however, to use an arrangement in which the optical axis is oriented at a diagonal relative to the optically transparent plate, and two vertical planes are disposed perpendicular to one another. For physical reasons, this arrangement is better suited than the classic stereo arrangement for determining the tilting angle (i.e., the diagonal positions of the object).
An easy-to-handle model object, such as a small model automobile on a 1:25 scale, serves as the hand-held input medium. The underside of the model object, that is, the side facing the camera(s), is provided with one or more flat marking symbols, such as letters, numbers or abstract symbols, for marking and identification purposes.
The camera images of these marking symbols are evaluated by a computer according to image-processing methods. The result of this evaluation is an identification of the model object, as well as its spatial position. In this way, a plurality of objects appearing simultaneously in the camera image(s) can be distinguished, and their individual positions can be indicated.
These results are transmitted to a graphics unit (graphics workstation PC graphics), with the associated objects, e.g., one, two or more vehicles of varying make, being displayed in their positions, as predetermined by the operator, on the display device of the graphics unit. Colored, flat marking symbols are advantageously applied to the underside of the model object. They can be located again (segmented) with simple means. In this case, the lighting must contain all of the color components (for example, visible white light). If the lighting is to remain invisible (for example, in spaces that are darkened for a more brilliant color-graphics display), the use of infrared light and retroreflecting marking symbols is advantageous.
The recognition of the model object and the determination of its position are subsequently effected through the separation of the shape and position parameters from the contour course of the marking symbols. The implementation of the method according to the invention is especially advantageous, because only one flat marking symbol is required for both partial tasks, namely recognition and localization, so the available surface can be well utilized, and little computation is required.
In addition to other alternatives, a barcode, for example, could be used for the recognition process, and a few simple, e.g., round, colored or reflective marks could additionally be used for ascertaining a position with classic methods of photogrammetry.